Some automated and semi-automated tools for linear feature extraction in two and three dimensions

ABSTRACT

A system for vector extraction comprising a vector extraction engine stored and operating on a network-connected computing device that loads raster images from a database stored and operating on a network-connected computing device, identifies features in the raster images, and computes a vector based on the features, and methods for feature and vector extraction.

CROSS-REFERENCE TO RELATED APPLICATIONS

Application No. Date Filed Title Current Herewith SOME AUTOMATED ANDSEMI- application AUTOMATED TOOLS FOR LINEAR FEATURE EXTRACTION IN TWOAND THREE DIMENSIONS Is a continuation of: 15/672,267 Aug. 8, 2017 SOMEAUTOMATED AND SEMI- AUTOMATED TOOLS FOR LINEAR FEATURE EXTRACTION IN TWOAND THREE DIMENSIONS which is a continuation of: 14/730,176 Jun. 3, 2015SOME AUTOMATED AND SEMI- Issued: Issue date: AUTOMATED TOOLS FOR LINEAR9,727,784 Aug. 8, 2017 FEATURE EXTRACTION IN TWO AND THREE DIMENSIONSwhich claims benefit of, and priority to: 62/007,079 Jun. 3, 2014TECHNIQUES FOR VECTOR EXTRACTION the entire specification of each ofwhich is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Art

The disclosure relates to the field of linear feature extraction,particularly from remotely-sensed raster data.

Discussion of the State of the Art

In the art of linear feature extraction, ROADTRACKER™ and similar toolsenable automated bulk extraction and semi-automated point-to-pointextraction of two-dimensional linear feature vectors fromremotely-sensed imagery. The extracted vectors represent centerlines oflinear features within the image raster. Extractions by these tools areimage-based, meaning the image content automatically drives the shapesof extracted vectors. In semi-automated extraction, the raster isdisplayed in a viewer and extraction is partially guided by user mouseclicks placed along a desired linear feature. Tools like theROADTRACKER™ can be used to extract centerlines for roads, trails, andhydrology features, and includes automatic smoothing of the vectors andautomatic topology cleaning (elimination of gaps (under-shoots) anddangles (over-shoots) where vectors are intended to be perfectlyincident to one another.) There are, however, several shortcomings tothese tools. One is that although the geometric accuracy of theautomated bulk extraction is usually good enough for isolated roads andcurved roads, it is often not satisfactory for rectangular city roadgrids because the extracted centerlines often are not as straight,parallel, or evenly-positioned as desired. Another deficiency is thatthe tools do not provide any capability for three-dimensional linearfeature extraction. And finally, when the semi-automated extraction of alinear feature involves a sequence of more than two mouse clicks,extraction does not commence until placement of the last mouse click. Amore preferred behavior would be for the extraction to growincrementally each time a new mouse click is added to the sequence.

What is needed are the following: a more accurate two-dimensionalautomated bulk extraction capability to capture rectangular city roadgrids; a three-dimensional automated and semi-automated linear featureextraction capability that utilizes a digital surface model (DSM) andperforms automatic vector smoothing and automatic topology cleaning; athree-dimensional automated and semi-automated linear feature vectorextraction capability that utilizes high-resolution stereo imagery andperforms automatic vector smoothing and automatic topology cleaning; andfinally, whether performing two-dimensional or three-dimensionalsemi-automated image-based linear feature vector extraction, when thefeature extraction involves a sequence of more than two mouse clicks,the extracted vector should grow incrementally each time a new mouseclick is added to the sequence.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, inpreferred embodiments, an interface and several methods for vectorextraction.

According to a preferred embodiment of the invention, a system forextracting two- and three-dimensional vectors comprising a vector serverstored and operating on a network-connected computing device, a rasterserver stored and operating on a network-connected computing device, adigital surface model (DSM) server stored and operating on anetwork-connected computing device, a vector extraction engine storedand operating on a network-connected computing device, and a renderingengine stored and operating on a network-connected computing device, isdisclosed. According to the embodiment, a vector server may retrievevectors from and send vectors to a vector storage such as a database orother data storage means (such as, for example, integral or removablehardware-based storage such as a hard disk drive, or software-basedstorage schema common in the art); a raster server may retrieve rasterimages from a raster storage, for example such as satellite images orsimilar raster image data that depict an actual physical environment; aDSM server may retrieve a DSM from a DSM storage, or may compute a DSMfrom the stereo disparity measurements of a stereo raster image pairretrieved from a raster storage. Retrieved vectors, rasters, and DSM maybe provided to a vector extraction engine, which under possibleadditional user inputs, may extract a plurality of new vectors withrespect to a raster image, DSM, and existing vectors, each new vectorcorrelating with a new linear feature in the raster image.

Vectors and rasters may then be provided to a rendering engine, that mayform a combined visualization of the two, showing how they relate toeach other, such as may be presentable on a viewer such as a displayscreen, for example for review by a human user. Additionally, a user mayinteract with the presented visualization using a variety of inputdevices such as (for example) a computer mouse or keyboard, such as tomanipulate the visualization or to indicate or guide where new vectorsare to be extracted within the raster or where undesirable existingvectors should be deleted. User input may be received by the renderingengine and utilized to update the rendering appropriately (such as tozoom in or out, for example), or may be further provided from therendering engine to the vector extraction engine as needed, for exampleto extract a new vector based on the user input. Newly-extracted vectorsmay be further provided to the vector server, for example to store thevectors for future reference.

According to another embodiment, a graphical user interface and methodsuch that in performing two-dimensional semi-automated image-basedlinear feature extraction (as an end in and of itself or as the basis ofa three-dimensional linear feature extraction), when the extractioninvolves a sequence of more than two mouse clicks from the user, theextracted vector grows incrementally each time a new mouse click isadded to the sequence, is disclosed.

According to another embodiment of the invention, a system and methodfor automated two-dimensional vector extraction of city road grids froman image raster, is disclosed.

According to another embodiment of the invention, a system and methodfor automated and semi-automated three-dimensional linear featureextraction from raster imagery and a DSM is disclosed. The extractionsrepresent the centerlines of the linear features as three-dimensionalvectors in the X, Y, Z coordinates of object space. The extractionincludes automated smoothing of the vectors and automated topologycleaning.

According to another embodiment of the invention, a system and methodfor automated and semi-automated three-dimensional linear featureextraction from high-resolution stereo imagery, is disclosed. Theextractions represent the centerlines of the linear features asthree-dimensional vectors in the X, Y, Z coordinates of object space.The extraction includes automated smoothing of the vectors and automatedtopology cleaning.

Image-Based Multi-Point Extraction Mode: As an additional embodiment, asemi-automated method for extracting two-dimensional vectors in themanner of “Image-Based Multi-Point Extraction Mode”, is disclosed. Inthis mode, in an initial step, the user may place a mouse-click atlocation P₁ in the a viewer. In subsequent steps, the user may placeadditional mouse clicks at locations P₂, P₃, . . . , P_(k−1) in theviewer. And in a final step, the user may indicate the last location inthe sequence, P_(k), with a double mouse click. In each step after theinitial step, after clicking at location P_(j+1), a vector extractionfrom P_(j) to P_(j+1) is computed (by the vector extraction engine) inreal-time (or near real-time) and displayed to the viewer. Thisextraction is realized as a least-cost path from P_(j) to P_(j+1)relative to a cost raster derived (possibly on-the-fly, possiblypre-computed) from the image raster in the viewer. While themouse-cursor location P_(j+1) is in motion, or while the least cost pathcomputation from P_(j) to P_(j+1) is not yet completed, the vector pathfrom P_(j) to P_(j+1) may be temporarily depicted in the viewer as astraight line segment. When the double-click occurs at location P_(k),the consecutive vector paths are concatenated by the vector extractionengine, and the resulting vector may be displayed on the raster in theviewer, and it may be committed to a persistent data store.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. It will beappreciated by one skilled in the art that the particular embodimentsillustrated in the drawings are merely exemplary, and are not to beconsidered as limiting of the scope of the invention or the claimsherein in any way.

FIG. 1 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in an embodiment of theinvention.

FIG. 2 is a block diagram illustrating an exemplary logical architecturefor a client device, according to an embodiment of the invention.

FIG. 3 is a block diagram showing an exemplary architectural arrangementof clients, servers, and external services, according to an embodimentof the invention.

FIG. 4 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

FIG. 5 is a block diagram of an exemplary system architecture for vectorextraction, according to a preferred embodiment of the invention.

FIG. 6 illustrates a general method for image-based multi-pointextraction, according to various embodiments disclosed herein.

FIG. 7A is an illustration of an exemplary graphical user interface forthree-dimensional vector extraction and monoscopic three-dimensionalviewing

FIG. 7B is an illustration of an exemplary graphical user interface forthree-dimensional vector extraction and stereoscopic three-dimensionalviewing

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, in a preferredembodiment of the invention, an interface and several methods for vectorextraction.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beappreciated that these are presented for illustrative purposes only andare not limiting of the inventions contained herein or the claimspresented herein in any way. One or more of the inventions may be widelyapplicable to numerous embodiments, as may be readily apparent from thedisclosure. In general, embodiments are described in sufficient detailto enable those skilled in the art to practice one or more of theinventions, and it should be appreciated that other embodiments may beutilized and that structural, logical, software, electrical and otherchanges may be made without departing from the scope of the particularinventions. Accordingly, one skilled in the art will recognize that oneor more of the inventions may be practiced with various modificationsand alterations. Particular features of one or more of the inventionsdescribed herein may be described with reference to one or moreparticular embodiments or figures that form a part of the presentdisclosure, and in which are shown, by way of illustration, specificembodiments of one or more of the inventions. It should be appreciated,however, that such features are not limited to usage in the one or moreparticular embodiments or figures with reference to which they aredescribed. The present disclosure is neither a literal description ofall embodiments of one or more of the inventions nor a listing offeatures of one or more of the inventions that must be present in allembodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or morecommunication means or intermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring non-simultaneously (e.g., because one step is described afterthe other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to one ormore of the invention(s), and does not imply that the illustratedprocess is preferred. Also, steps are generally described once perembodiment, but this does not mean they must occur once, or that theymay only occur once each time a process, method, or algorithm is carriedout or executed. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described herein, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described herein, it will be readily apparent that a singledevice or article may be used in place of the more than one device orarticle.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should beappreciated that particular embodiments may include multiple iterationsof a technique or multiple instantiations of a mechanism unless notedotherwise. Process descriptions or blocks in figures should beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process. Alternate implementations areincluded within the scope of embodiments of the present invention inwhich, for example, functions may be executed out of order from thatshown or discussed, including substantially concurrently or in reverseorder, depending on the functionality involved, as would be understoodby those having ordinary skill in the art.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be describedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, or other appropriatecomputing device), a consumer electronic device, a music player, or anyother suitable electronic device, router, switch, or other suitabledevice, or any combination thereof. In at least some embodiments, atleast some of the features or functionalities of the various embodimentsdisclosed herein may be implemented in one or more virtualized computingenvironments (e.g., network computing clouds, virtual machines hosted onone or more physical computing machines, or other appropriate virtualenvironments).

Referring now to FIG. 1, there is shown a block diagram depicting anexemplary computing device 100 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 100 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 100 may be adaptedto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 100 includes one or more centralprocessing units (CPU) 102, one or more interfaces 110, and one or morebusses 106 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 102may be responsible for implementing specific functions associated withthe functions of a specifically configured computing device or machine.For example, in at least one embodiment, a computing device 100 may beconfigured or designed to function as a server system utilizing CPU 102,local memory 101 and/or remote memory 120, and interface(s) 110. In atleast one embodiment, CPU 102 may be caused to perform one or more ofthe different types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 102 may include one or more processors 103 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 103 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 100. In a specificembodiment, a local memory 101 (such as non-volatile random accessmemory (RAM) and/or read-only memory (ROM), including for example one ormore levels of cached memory) may also form part of CPU 102. However,there are many different ways in which memory may be coupled to system100. Memory 101 may be used for a variety of purposes such as, forexample, caching and/or storing data, programming instructions, and thelike. It should be further appreciated that CPU 102 may be one of avariety of system-on-a-chip (SOC) type hardware that may includeadditional hardware such as memory or graphics processing chips, such asa Qualcomm SNAPDRAGON™ or Samsung EXYNOS™ CPU as are becomingincreasingly common in the art, such as for use in mobile devices orintegrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 110 are provided as network interfacecards (NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 110 may forexample support other peripherals used with computing device 100. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 110 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity AN hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 1 illustrates one specificarchitecture for a computing device 100 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 103 may be used, and such processors 103may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 103 handlescommunications as well as routing computations, while in otherembodiments a separate dedicated communications processor may beprovided. In various embodiments, different types of features orfunctionalities may be implemented in a system according to theinvention that includes a client device (such as a tablet device orsmartphone running client software) and server systems (such as a serversystem described in more detail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 120 and local memory 101) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 120 or memories101, 120 may also be configured to store data structures, configurationdata, encryption data, historical system operations information, or anyother specific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a Java™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 2,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 200 includes processors210 that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 230. Processors 210 may carry out computinginstructions under control of an operating system 220 such as, forexample, a version of Microsoft's WINDOWS™ operating system, Apple's MacOS/X or iOS operating systems, some variety of the Linux operatingsystem, Google's ANDROID™ operating system, or the like. In many cases,one or more shared services 225 may be operable in system 200, and maybe useful for providing common services to client applications 230.Services 225 may for example be WINDOWS™ services, user-space commonservices in a Linux environment, or any other type of common servicearchitecture used with operating system 210. Input devices 270 may be ofany type suitable for receiving user input, including for example akeyboard, touchscreen, microphone (for example, for voice input), mouse,touchpad, trackball, or any combination thereof. Output devices 260 maybe of any type suitable for providing output to one or more users,whether remote or local to system 200, and may include for example oneor more screens for visual output, speakers, printers, or anycombination thereof. Memory 240 may be random-access memory having anystructure and architecture known in the art, for use by processors 210,for example to run software. Storage devices 250 may be any magnetic,optical, mechanical, memristor, or electrical storage device for storageof data in digital form (such as those described above, referring toFIG. 1). Examples of storage devices 250 include flash memory, magnetichard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 3, there is shown a blockdiagram depicting an exemplary architecture 300 for implementing atleast a portion of a system according to an embodiment of the inventionon a distributed computing network. According to the embodiment, anynumber of clients 330 may be provided. Each client 330 may run softwarefor implementing client-side portions of the present invention; clientsmay comprise a system 200 such as that illustrated in FIG. 2. Inaddition, any number of servers 320 may be provided for handlingrequests received from one or more clients 330. Clients 330 and servers320 may communicate with one another via one or more electronic networks310, which may be in various embodiments any of the Internet, a widearea network, a mobile telephony network (such as CDMA or GSM cellularnetworks), a wireless network (such as WiFi, Wimax, LTE, and so forth),or a local area network (or indeed any network topology known in theart; the invention does not prefer any one network topology over anyother). Networks 310 may be implemented using any known networkprotocols, including for example wired and/or wireless protocols.

In addition, in some embodiments, servers 320 may call external services370 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 370 may take place, for example, via one or morenetworks 310. In various embodiments, external services 370 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 230 are implemented on a smartphone or other electronicdevice, client applications 230 may obtain information stored in aserver system 320 in the cloud or on an external service 370 deployed onone or more of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 330 or servers 320 (orboth) may make use of one or more specialized services or appliancesthat may be deployed locally or remotely across one or more networks310. For example, one or more databases 340 may be used or referred toby one or more embodiments of the invention. It should be understood byone having ordinary skill in the art that databases 340 may be arrangedin a wide variety of architectures and using a wide variety of dataaccess and manipulation means. For example, in various embodiments oneor more databases 340 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, Hadoop Cassandra, Google BigTable, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 360 and configuration systems 350. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 360 orconfiguration system 350 or approach is specifically required by thedescription of any specific embodiment.

FIG. 4 shows an exemplary overview of a computer system 400 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 400 withoutdeparting from the broader scope of the system and method disclosedherein. CPU 401 is connected to bus 402, to which bus is also connectedmemory 403, nonvolatile memory 404, display 407, I/O unit 408, andnetwork interface card (NIC) 413. I/O unit 408 may, typically, beconnected to keyboard 409, pointing device 410, hard disk 412, andreal-time clock 411. NIC 413 connects to network 414, which may be theInternet or a local network, which local network may or may not haveconnections to the Internet. Also shown as part of system 400 is powersupply unit 405 connected, in this example, to ac supply 406. Not shownare batteries that could be present, and many other devices andmodifications that are well known but are not applicable to the specificnovel functions of the current system and method disclosed herein. Itshould be appreciated that some or all components illustrated may becombined, such as in various integrated applications (for example,Qualcomm or Samsung SOC-based devices), or whenever it may beappropriate to combine multiple capabilities or functions into a singlehardware device (for instance, in mobile devices such as smartphones,video game consoles, in-vehicle computer systems such as navigation ormultimedia systems in automobiles, or other integrated hardwaredevices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

Conceptual Architecture

FIG. 5 is a block diagram illustrating an exemplary system 500 forvector extraction, according to a preferred embodiment of the invention.As illustrated, a vector extraction engine 501 may be stored andoperated on a network-connected computing device such as a computerserver or workstation As illustrated, vector extraction engine 501 maybe connected to a database 502 for raster storage, for example to storeand retrieve raster images for vector extraction, and a database 503 forvector storage, such as to store and retrieve extracted vectorinformation. It should also be appreciated that multiple or singulardatabases may be employed according to the embodiment, for examplestoring both raster and vector information in a singular data storage.

As further illustrated, a vector display 504 may be used to provide avisual output to a user, for example, to enable user review and tofacilitate user interaction via input devices 505 such as to indicate anew linear feature that is to be extracted from a raster image. In thismanner, user input is received by the rendering engine and sent to thevector extraction engine 501. Additionally, a digital surface model(DSM) database 506 may be utilized to store and provide DSM informationto a DSM server 507, which may provide DSM data for use in vectoroperations.

It should be appreciated that a variety of connections and interactionsmay be possible according to the embodiment, such as via a datacommunication network such as the Internet, facilitating a distributedarrangement where the illustrated components need not necessarily belocated within physical proximity of each other, for example in acloud-based or software as a service (SaaS) arrangement providing thevector extraction utility of the invention to connected users.

Detailed Description of Exemplary Embodiments

FIG. 6 illustrates a general method 800 for image-based semi-automatedmulti-point extraction of two-dimensional vectors, according to variousembodiments disclosed herein. In an initial step 801, a user may clickon a first point in a viewer, and in a next step 802 may then click on anew point. In a next step 803 the vector extraction engine mayautomatically build a least cost path between the two points. Suchprocedure may continue iteratively wherein a user continues clickingadditional points and for each new point clicked, the vector extractionengine automatically builds a least cost path between the new point andthe previous point. In a next step 804 a user may “double-click” on apoint to indicate it as a final point, and in a final step 805 the leastcost paths may be concatenated into a single vector.

Two-Dimensional Linear Feature Extraction of City Road Grids

In all the variations of two-dimensional automated city road gridextraction described below, existing technology may be utilized toperform an initial automated bulk “raw” two-dimensional vectorextraction of road centerlines from remotely sensed imagery.

Below are described several exemplary variations on the method ofextracting city road grids. The user interface is a viewer showing theraster image of interest. The user can interact with this viewer viainput devices (e.g., mouse and keyboard) and graphical tools.

Variation 1: User specifies a constraint region delimiting the desiredroad grid in the raster image in the viewer, and specifies a linesegment along one of the roads so as to indicate one of the two roaddirections in the extraction. The other road direction will be assumedperpendicular to this line segment. Road grid vectors will beautomatically extracted from within the constraint region by the vectorextraction engine as follows: Perform initial automatic bulk “raw”vector extraction of road centerlines (e.g., using RoadTrackertechnology) within the constraint region of the raster image; identifylengthy portions of these vectors where each such portion can be fittightly (in a least squares sense) with a straight line segment; removevector portions that are short or that cannot be so fit; remove linesegments that are not sufficiently parallel to the user-designated linesegment or its perpendicular; identify line segments that belong to thesame road based on line segment direction and position; for linesegments deemed to be along the same road, fit them with a line segmentparallel to the user-designated line segment or its perpendicular. Thiscompletes the extraction. The result is displayed in the viewer. Theextraction can be saved off to persistent data store.

Variation 2: User specifies a constraint region delimiting the desiredroad grid in the raster image in the viewer. Road grid vectors will beautomatically extracted from within the constraint region by the vectorextraction engine as follows: Perform initial automatic bulk “raw”vector extraction of road centerlines within the constraint region ofthe raster image; identify lengthy portions of these vectors where eachsuch portion can be fit tightly (in a least squares sense) with astraight line segment; remove vector portions that are short or thatcannot be so fit; identify the line segments that belong to the sameroad, based on line segment direction and position; for line segmentsdeemed to be along the same road, fit them with a line segment; let S₁denote a maximal set of line segments that are more or less parallel andlet S₂ denote a maximal set of line segments that are more or lessparallel to each other and more or less perpendicular to S₁; identify asingle average or median representative road direction for S₁ and asingle average or median representative road direction for S₂, such thatthe two representative directions are perpendicular to each other;rotate each line segment in S₁ about its center point so that it pointsalong the representative direction for S₁, and do similarly for the linesegments in S₂. This completes the extraction. The result is displayedin the viewer. The extraction can be saved off to persistent data store.

Variation 3: User additionally designates that for one road direction inthe grid, the extracted road vectors should be not only parallel, butequally-spaced as well. Road grid vectors will be automaticallyextracted from within the constraint region by the vector extractionengine as follows: First extract initial road grid vectors usingVariation 1 or 2 above. Compute the distance between each pair ofconsecutive roads that parallel to the user-designated road directionand notionally plot the results as points on a horizontal number line.Among these points, identify the cluster that contains the most values,or alternatively, the cluster that contains the smallest value. Computethe median M of this cluster—this value will be taken as the evenspacing between roads for the user-designated road direction. Use astandard minimization technique to determine minimal parallel offsets ofthe road vectors in the initial extraction so that the resulting roadvectors are evenly-spaced in increments of M. The minimization may useany of the following objective functions: sum squared residuals, sum ofresidual magnitudes, or maximum residual. This completes the extraction.The result is displayed in the viewer. The extraction can be saved offto persistent data store.

Extracted three-dimensional linear feature vectors will be expressed asthree-dimensional vectors in the X, Y, Z coordinates of object space.The extraction will include the automated smoothing of the vectors aswell as automated topology cleaning.

According to another embodiment of the invention, FIG. 7A is a diagramillustrating a monoscopic graphical user interface 700 tied to a vectorextraction engine (the latter not shown in the figure): enabling theuser to extract three-dimensional linear feature vectors correspondingto the centerlines of three-dimensional linear features given by araster image and DSM; enabling the user to view extracted vectorsagainst a digital surface model (DSM) for three-dimensional context;enabling the user in semi-automated mode to guide the extraction of aparticular linear feature, for example using connected mouse andkeyboard 701; enabling the user to commit extracted vectors to apersistent data store; enabling the user to delete undesirable extractedvectors. The system may incorporate two-dimensional image-based andnon-image-based linear feature extraction as part of the process ofextracting three-dimension linear feature vectors. The user interface700 may comprise a plurality of synchronized graphical viewers such as:

-   -   A three-dimensional viewer 710 that may display the        three-dimensional vectors 711 against a digital surface model        (DSM) 712 or, optionally, against an empty three-dimensional        object space. The viewer may optionally offer perspective or        non-perspective viewing, and enable the user to pan, zoom, and        yaw about the line of sight. The DSM 712 may be opaque or        semi-transparent or represented as a wireframe. A        semi-transparent DSM 712 may allow a user to see where a        three-dimensional vector 711 lies above or below the landscape        represented by the DSM 712.    -   A two-dimensional XY-viewer 720 may display the        three-dimensional vectors 711 projected vertically to the        XY-plane 721 of a raster image. The viewer may, as above,        support pan, zoom, and yaw about the line of sight. Projected        vectors 711 as shown in the viewer 720 may be user-selectable,        for example, to indicate corresponding three-dimensional vectors        to be deleted.

For automated bulk image-based three-dimensional linear featureextraction, a vector extraction engine first extracts two-dimensionalXY-vectors from the raster image via, say, the automated version of theROADTRACKER™ (which also takes care of smoothing and topology cleaningof extracted two-dimensional vectors.) These extracted two-dimensionalvectors may then be displayed in the XY-image viewer. The vectorextraction engine then automatically projects each two-dimensionalXY-vector vertically along the Z-axis to the DSM (or slightly above it),and automatically smooths it in the Z-dimension. For the most part, theresulting three-dimensional vector may ride slightly above the DSM. Careis taken by the vector extraction engine to ensure that whentwo-dimensional extracted vectors are incident in the XY plane, theirprojected, three-dimensional, smoothed versions are incident asthree-dimensional vectors. Thus the connection topology of thethree-dimensional vectors is the same as that of the two-dimensionalvectors. The extracted three-dimensional vectors may be displayed in thethree-dimensional viewer.

For semi-automated image-based three-dimensional linear featureextraction, the user may designate, with mouse clicks, two or morepoints on the image raster in the XY-Viewer. A vector extraction enginemay first extract the vector two-dimensionally (the extracted vectorpassing through all the user-designated mouse-click points) inimage-based fashion from the image raster via, say, the semi-automatedversion of the ROADTRACKER™. The extracted two-dimensional vector maythen be displayed in the XY image viewer. The vector extraction enginethen automatically projects the newly extracted two-dimensional XYvector vertically along the Z axis to the DSM (or slightly above it) andautomatically smooths it in the Z-dimension. For the most part, theresulting three-dimensional vector may ride slightly above the DSM. Careis taken by the vector extraction engine to ensure that whenever thetwo-dimensional version of the new vector is incident to anothertwo-dimensional vector in the XY plane, the three-dimensional versionsof both vectors are also incident. Thus the connection topology of thethree-dimensional vectors remains the same as that of thetwo-dimensional vectors. The extracted three-dimensional vector may bedisplayed in the three-dimensional viewer.

According to another embodiment of the invention, FIG. 7B is a diagramillustrating a stereoscopic graphical user interface 730 tied to avector extraction engine (the latter not shown in the figure): enablingthe user to extract three-dimensional linear feature vectorscorresponding to the centerlines of three-dimensional linear featuresgiven by a stereo raster image pair; enabling the user to view extractedvectors in a three-dimensional stereo context; enabling the user insemi-automated mode to guide the extraction of a particular linearfeature via, say, mouse clicks; enabling the user to commit extractedvectors to a persistent data store; enabling the user to deleteundesirable extracted vectors. The system may incorporatetwo-dimensional image-based and non-image-based linear featureextraction as part of the process of extracting three-dimension linearfeature vectors. The user interface 730 may comprise a plurality ofgraphical viewers such as:

-   -   A stereo three-dimensional viewer 740 displaying the        three-dimensional vectors against a (possibly semi-transparent)        stereo view of the landscape. It may be possible for a user to        see where a vector lies above or below the perceived landscape.        The viewer supports pan, zoom, and yaw about line of sight.    -   A monoscopic XY-image viewer 720, showing an orthorectified        version of one of the raster images of the stereo image pair.        (The manner of the orthorectification is explained below.) The        viewer may enable the user to pan, zoom, and yaw about line of        sight. Extracted two-dimensional vectors in the viewer may be        user-selectable, for example, to indicate that their        corresponding three-dimensional vectors are to be deleted.

In the stereo extraction system, orthorectification of the raster imagein the XY image viewer may be based on a digital surface model (DSM)automatically constructed from the stereo image pair via automaticallycomputed stereo disparity measurements. If we are dealing withhigh-resolution stereo imagery, the resulting DSM will have reasonablegeospatial accuracy.

In the stereo viewing and extraction system, the aforementionedorthorectified raster and DSM may be employed in the same way as in themonoscopic viewing and extraction system in that after automated orsemi-automated two-dimensional extraction from the orthorectified rasterin the XY plane is performed, the extracted two-dimensional vectors areprojected vertically along the Z-axis to the DSM. In the stereo system,the two-dimensional extracted vectors may be displayed in the monoscopicXY viewer and the corresponding three-dimensional vectors may bedisplayed in the stereo viewer.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for three-dimensional vector extractionfrom imagery of a surface, comprising: a vector extraction engineoperating on a computing device or network of computing devices; whereinthe vector extraction engine: builds a cost raster; builds at least onedigital surface model raster based at least in part on the cost raster;combines the rasters to form a new three-dimensional cost raster;identifies image features in at least one of the rasters; computesinstantaneous values for the identified features; calculates a vectorbased at least in part on the instantaneous values; and projects thevector onto the new three-dimensional raster.
 2. A method forthree-dimensional vector extraction from image data of a surface,comprising the steps of: building, using a vector extraction engineoperating on a computing device or network of computing devices, a costraster; building at least one digital surface model raster based atleast in part on the cost raster; and combining the rasters to form anew three-dimensional cost raster; identifying, using a vectorextraction engine stored and operating on a workstation computer, imagefeatures in at least one of the rasters; computing instantaneous valuesfor the identified features; calculating a vector based at least in parton the instantaneous values; and projecting the vector onto thethree-dimensional raster.